Apparatus and method for insertion and deletion in multi-dimensional to linear address space translation

ABSTRACT

A translation system can translate a storage request to a physical address using fields as keys to traverse a map of nodes with node entries. A node entry can include a link to a next node or a physical address. Using a portion of the key as noted in node metadata, a node entry can be determined. When adding node entries to a node, a node utilization can exceed a threshold value. A new node can be created such that node entries are split between the original and new node. Node metadata of the parent node, new node and original node can be revised to identify which parts of the key are used to identify a node entry. When removing node entries from a node, node utilization can cross a minimum threshold value. Node entries from the node can be merged with a sibling, or the map can be rebalanced.

CROSS-REFERENCES TO RELATED APPLICATIONS

This application claims the benefit of and incorporates by reference forall purposes the full disclosure of the following patent applications intheir entirety:

-   U.S. Provisional Patent Application No. 61/794,647, filed Mar. 15,    2013 titled “APPARATUS AND METHOD FOR TRANSLATION FROM    MULTI-DIMENSIONAL TO LINEAR ADDRESS SPACE IN STORAGE,”-   U.S. Provisional Patent Application No. 61/798,754, filed Mar. 15,    2013 titled “HIGH DENSITY SERVER STORAGE UNIT,”-   U.S. Provisional Patent Application No. 61/788,613, filed Mar. 15,    2013 titled “COMPRESSOR RESOURCES FOR HIGH DENSITY STORAGE UNITS,”-   U.S. Provisional Patent Application No. 61/793,141, filed Mar. 15,    2013 titled “MASS STORAGE DEVICE AND METHOD OF OPERATING THE SAME TO    BACK UP DATA STORED IN VOLATILE MEMORY,”-   U.S. Provisional Patent Application No. 61/793,591, filed Mar. 15,    2013 titled “MASS STORAGE DEVICE AND METHOD OF OPERATING THE SAME TO    STORE PARITY DATA,” and-   U.S. Provisional Patent Application No. 61/799,023, filed Mar. 15,    2013 titled “VERTICALLY INTEGRATED STORAGE.”

The following patent applications, concurrently filed with the presentapplication, are incorporated herein by reference in their entirety:

-   Patent Application titled “APPARATUS AND METHOD FOR TRANSLATION FROM    MULTI-DIMENSIONAL TO LINEAR ADDRESS SPACE IN STORAGE,” Attorney    docket number: 95469-878915-000310US.-   Patent Application titled “APPARATUS AND METHOD FOR REFERENCING    DENSE AND SPARSE INFORMATION IN MULTI-DIMENSIONAL TO LINEAR ADDRESS    SPACE TRANSLATION,” Attorney docket number: 95469-890619-000320US.-   Patent Application titled “APPARATUS AND METHOD FOR USING FIELDS IN    N-SPACE TRANSLATION OF STORAGE REQUESTS,” Attorney docket number:    95469-890620-000330US.-   Patent Application titled “APPARATUS AND METHOD FOR CLONING AND    SNAPSHOTTING IN MULTI-DIMENSIONAL TO LINEAR ADDRESS SPACE    TRANSLATION,” Attorney docket number: 95469-890628-000350US.

BACKGROUND

Aspects of the disclosure relate to computing and communicationtechnologies. In particular, aspects of the disclosure relate tosystems, methods, apparatuses, and computer-readable media for improvingperformance of storage devices.

Storage devices for enterprise systems require massive storage capacity.Additionally, storage solutions for enterprise systems requiresophisticated storage techniques for reliability, robustness, faulttolerance, maximizing storage capacity, minimizing power consumption,and reducing latency. Various storage industry players have specializedin aspects of these storage techniques in a segmented manner providingpiecemeal solutions. Combining of these various segmented solutionsresults into a clunky storage solution results in a solution that isless than the sum of its parts and significantly underperforms acrossthe board. The segmentation and underperformance of the availablesolutions today results in a significant deterrent in adaptation ofnewer storage technologies, such as solid state devices.

BRIEF SUMMARY

Certain embodiments of the present invention relate to translating astorage request having multiple fields to a physical address using thefields as keys to traverse a map table. By using a map table, multiplestorage services can be condensed into a single map traversal. A map canbe constructed of root nodes, inner nodes and leaf nodes. The rootnodes, inner nodes and leaf nodes can include entries that can beindexed or sorted by a field or part of a field. The entries of rootnodes and inner nodes can also include pointers to a next node. Leafnode entries can include values, such as a physical address and/or otherattributes (e.g. ECC values, etc.). When traversing the storage map, thetranslation system can start at a root node. Using one or more fieldsfrom the request, the translation system can determine entries in theroot node and/or inner nodes that have a pointer to a next node. Thepointers in the determined entries can be followed until a leaf node isfound. Using one or more fields from the request or portions thereof, anentry in the leaf node can be determined. The values stored in thedetermined leaf node can then be returned, such as the physical addressand/or other attributes. For example, a read storage request of logicalunit number (LUN), logical block address (LBA) and snapshot number(SNAP) (i.e. Read(LUN, LBA, SNAP)) can be processed by traversing astorage map to return a physical address and a length attribute of 240(i.e. return(PA, L)).

A closest match algorithm can be used to retrieve unchanged databelonging to a prior generation of data. For example, when a snapshot iscreated, a snapshot number can be incremented. In the embodiment,changes are not needed to the underlying snapshot data. When searchingfor the new snapshot, the process of selecting a LUN and followingpointers within the inner nodes can be performed as described above.However, the new snapshot can be devoid of any data. In the this case ofa snapshot that does not have new data at the location requested, thetranslation system can select an entry that has the same LUN and LBA,but has a closest earlier snapshot entry. This embodiment ofsnapshotting allows for the deduplication of information over multiplegenerations of snapshots. This embodiment of snapshotting can provide afast method of taking a snapshot, while preventing a loss of time due tocopying.

The storage map can also be optimized for dense and sparse information.An entry in a node can be stored in a hashed storage area of a node or asorted area of a node. In the hashed storage area, a key (e.g. a fieldof a storage request) can be used to calcuate an index into the hashedstorage area. A hashed storage area, in some embodiments, does not storethe key. The hashed storage area can service dense key ranges, wherekeys are more bunched together than spread out. The hashed area canprovide a a constant lookup time. In the sorted area of a node, a keycan be used to search the sorted area for a matching entry. In someembodiments, the entries are linearly sorted and a binary search is usedto determine the location of an entry having a matching key. A sortedstorage area, in some embodiments, stores the key to allow for thecomparision of keys during the search. The sorted storage area canservice sparse key ranges, where keys are more spread out than bunchedtogether. The sorted storage area can provide a compact space forstorage of sparse keys. Storage maps and/or individual nodes can be ofmultiple types including only hashed, only sorted or a hybrid of both.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects of the disclosure are illustrated by way of example. In theaccompanying figures, like reference numbers indicate similar elements,and:

FIG. 1 shows a simplified diagram of a layered approach for accessingstorage hardware in accordance with at least one embodiment.

FIG. 2 shows a simplified diagram of a vertically integrated approachfor accessing storage hardware in accordance with at least oneembodiment.

FIG. 3 shows an illustrative example of a system for translating storagerequests into physical addresses in accordance with at least oneembodiment.

FIG. 4 shows a diagram of a node structure for translating storagerequests into physical addresses in accordance with at least oneembodiment.

FIG. 5 shows a diagram of translating a storage request into physicaladdresses using a closest match snapshot in accordance with at least oneembodiment.

FIG. 6 shows a diagram of a map for translating storage requests intophysical addresses in accordance with at least one embodiment.

FIG. 7A shows a diagram of a map for translating storage requests intophysical addresses in accordance with at least one embodiment.

FIG. 7B shows a diagram of a map for translating storage requests intophysical addresses after an update in accordance with at least oneembodiment.

FIG. 7C shows a diagram of a map for translating storage requests intophysical addresses after update and split in accordance with at leastone embodiment.

FIG. 8A shows a diagram of a map for translating storage requests intophysical addresses in accordance with at least one embodiment.

FIG. 8B shows a diagram of a map for translating storage requests intophysical addresses after three deletions in accordance with at least oneembodiment.

FIG. 8C shows a diagram of a map for translating storage requests intophysical addresses after three deletions and a merge in accordance withat least one embodiment.

FIG. 9 shows a diagram of a map for translating storage requests intophysical addresses using cloning information in accordance with at leastone embodiment.

FIG. 10 illustrates an example of a computing system in which one ormore embodiments may be implemented.

FIG. 11 shows an illustrative example of a process that may be used tosearch a map for translating a storage request into physical addressesin accordance with at least one embodiment.

FIG. 12 shows an illustrative example of a process that may be used toupdate a map for translating a storage request into physical addressesin accordance with at least one embodiment.

FIG. 13 shows an illustrative example of a process that may be used todelete an entry in a map for translating a storage request into physicaladdresses in accordance with at least one embodiment.

FIG. 14 shows an illustrative example of a process that may be used toclone a Logical Unit in a map for translating a storage request intophysical addresses in accordance with at least one embodiment.

FIG. 15 shows an illustrative example of a process that may be used tosnapshot a Logical Unit in a map for translating a storage request intophysical addresses in accordance with at least one embodiment.

DETAILED DESCRIPTION

Several illustrative embodiments will now be described with respect tothe accompanying drawings, which form a part hereof. While particularembodiments, in which one or more aspects of the disclosure may beimplemented, are described below, other embodiments may be used andvarious modifications may be made without departing from the scope ofthe disclosure or the spirit of the appended claims.

Certain embodiments of the present invention relate to translating astorage request having multiple fields to a physical address using thefields as keys to traverse a map table. By using a map table, multiplestorage services can be condensed into a single map traversal. A map canbe constructed of root nodes, inner nodes and leaf nodes. The rootnodes, inner nodes and leaf nodes can include entries that can beindexed or sorted by a field or part of a field. The entries of rootnodes and inner nodes can also include pointers to a next node. Leafnode entries can include values, such as a physical address and/or otherattributes (e.g. ECC values, etc.). When traversing the storage map, thetranslation system can start at a root node. Using one or more fieldsfrom the request, the translation system can determine entries in theroot node and/or inner nodes that have a pointer to a next node. Thepointers in the determined entries can be followed until a leaf node isfound. Using one or more fields from the request or portions thereof, anentry in the leaf node can be determined. The values stored in thedetermined leaf node can then be returned, such as the physical addressand/or other attributes.

For example, a read storage request of logical unit number, logicalblock address and snapshot number (i.e. Read(LUN, LBA, SNAPSHOT)) can beprocessed by traversing a storage map to return a physical address and alength attribute of 240 (i.e. return(PA, L)). In this example, a rootnode is sorted by LUN; two inner nodes are sorted by LBA and a leaf nodeis sorted by snapshot. After receiving the read request, a translationsystem can start at the root node. The translation system can thenselect the entry for the LUN and follow a pointer to a first inner node.The header of the first inner node can contain instructions for thetranslation system to compute a key for an entry in the first inner nodethrough use of a modulus of a first portion of the LBA. The selectedentry of the first inner node can contain a pointer to a second innernode. The pointer can then be followed to a second inner node. Theheader of the second inner node can contain instructions for thetranslation system to compute a key to a second entry in the secondinner node by using a modulus of a second portion of the LBA. The secondentry can contain a pointer to a leaf node. After following the secondentry to the leaf node, the translation system can be instructed to findthe snapshot. Using the key described in the header of the leaf node,the snapshot number is used to find the key in the leaf node. Once thekey is found in a leaf entry, the corresponding PA and L values can beretrieved from the leaf entry and returned.

A closest match algorithm can be used to retrieve unchanged databelonging to a prior generation of data. For example, when a snapshot iscreated, a snapshot number can be incremented. In the embodiment,changes are not needed to the underlying snapshot data. When searchingfor the new snapshot, the process of selecting a LUN and followingpointers within the inner nodes can be performed as described above.However, the new snapshot can be devoid of any data. In the this case ofa snapshot that does not have new data at the location requested, thetranslation system can select an entry that has the same LUN and LBA,but has a closest earlier snapshot entry. This embodiment ofsnapshotting allows for the deduplication of information over multiplegenerations of snapshots. This embodiment of snapshotting also providesa fast method of taking a snapshot, while preventing a loss of time dueto copying.

When data is written to the new snapshot, the new data can be stored ina new entry. When data in the snapshot is overwritten, the data can betraced to a leaf node. An entry with new snapshot number and the old LUNand LBA data can be created in the leaf node. This method of storageallows the translation system to traverse the storage map to find a LUN,LBA and snapshot value, but use a closest snapshot value if an exactsnapshot value cannot be found.

The storage map can also be optimized for dense and sparse information.An entry in a node can be stored in a hashed storage area of a node or asorted area of a node. In the hashed storage area, a key (e.g. a fieldof a storage request) can be used to calcuate an index into the hashedstorage area. A hashed storage area, in some embodiments, does not storethe key. The hashed storage area can service dense key ranges, wherekeys are more bunched together than spread out. In the sorted area of anode, a key can be used to search the sorted area for a matching entry.In some embodiments, the entries are linearly sorted and a binary searchis used to determine the location of an entry having a matching key. Asorted storage area, in some embodiments, stores the key to allow forthe comparision of keys during the search. The sorted storage area canservice sparse key ranges, where keys are more spread out than bunchedtogether. The sorted storage area can provide a compact space forstorage of sparse keys. Storage maps and/or individual nodes can be onlyhashed, only sorted or a hybrid of both.

FIG. 1 illustrates a simplified diagram of a layered approach foraccessing storage hardware. The layered approach for storage devicesuses a number of stacked software/firmware layers for accessing thestorage hardware from the application layer 116. As shown in FIG. 1, Inone implementation, the layered approach includes the firmwareassociated with the storage device 118, Redundant Array of IndependentDisks (RAID) layer 104, compression layer 106, deduplication layer 108,snapshots/clones/thin provisioning layer 110, file system layer 112, OSinterface layer 114, and application layer 116. The firmware interactingwith the hardware may also act as another layer 102 implementing its ownmanagement 120, caching 122, journaling 124, mapping 126, andwrite/garbage collection 128. The various layers in the stack may bedeveloped by various storage device industry vendors.

The RAID software/firmware layer 104 provides fault tolerance byspreading the data and parity information across multiple disks orplanes. The compression layer 106 compresses data allowing for efficientand faster access of storage medium. The deduplication layer 108generally generates fingerprints using hash functions for each commandthat a host issues to the storage device. The deduplication layer 108detects duplication by comparing the current generated fingerprint withthe maintained ones. In one implementation, the deduplication layer 108maps the duplicate blocks from the various linear addresses to the samephysical address, reducing the number of writes to storage and using thestorage space more efficiently. The file system layer 112 providesabstraction for storing, retrieving and updating files on the storagedevice. Additionally, the file system manages access to data andmetadata of the files and available space on the device. The OSinterface layer 114 provides the application layer 116 a standardizedinterface for interacting with the storage device by calling functioncalls enabled by the OS interface layer 114.

In addition to their primary roles discussed above, most of the layersof the storage stack also perform additional house-keeping routines,such as maintaining memory, management functions, caching, linear tophysical address mapping, garbage collection and journaling of statesfor protection against catastrophic events. Garbage collection may referto the releasing of memory/storage resources no longer needed by thelayer. Journaling may refer to logging state before committing the statein state machine. In the event of a catastrophic event, such as a systemcrash or a power failure, journaling may enable the system to recoverfaster and avoid corruption of system state.

Many of these house-keeping routines are duplicated in each layer of thestorage stack, since these house-keeping routines performed by eachlayer are dedicated to that specific layer and isolated from the otherlayers because of the layered architecture causing significant memory,processing and performance overhead.

Furthermore, for an application from the application layer 116 tocommunicate with the storage device 102, the message must pass throughseven layers as shown in FIG. 1. The passing of the data message throughmultiple layers requires a number of encapsulation and de-encapsulationsteps that also generates significant overhead.

The interface between each layer also creates bottlenecks. Moreover, theinterface abstracts away details and allows for only limited visibilityto the next layer below and beyond requiring duplication of functions inthe software stack, such as compression and journaling of state. Forexample, the file system layer 112, the Snapshots/clones thinprovisioning layer 110 and the deduplication layer may all implementcompression algorithms. However, once data is compressed there is verylittle benefit in repeatedly compressing data, resulting in wastedresources, in terms of latency and performance. Therefore, duplicationof functions results in processing and memory overhead considerablydragging down the performance of the system.

Each layer also manages its own mapping to translate the message fromone layer to another. Mapping operations are expensive operations,increasing latency of data operations and degrading the performance ofthe system even further.

Moreover, the storage stack layers are developed by different vendorsand adhere to various standard bodies. Every layer is developed inisolation from the other layers in the storage stack software vastlyrepeating the same functionality in different manifestationssignificantly increasing the probability of bugs in the system.Additionally, the storage stack layered approach hampers innovation inthe product line, since any innovation that disturbs the interfacesbetween the different layers goes through a complex negotiation processwith the various stake holders, such as the vendors for the differentlayers in the software stack. Furthermore, the performance degradationhas a multiplicative in the layered architecture further exasperatingperformance issues.

The many repeated tasks and functions shown in FIG. 1 can be condensedinto a storage system that can reduce the redundancy shown in FIG. 1.FIG. 2 illustrates a simplified diagram of a vertically integratedapproach for accessing storage hardware. A client 206 that includes anapplication 116 and operating system interface 114 can access a storagesystem 200. The storage system 200 can include a management system 204for managing storage hardware 202 that can include solid state drivehardware 130. The management system can be configured to provide aRedundant Array of Independent Disks (RAID) layer 104, compression layer106, deduplication layer 108, Snapshots/clones/thin provisioning layer110 and file system layer 112. Vertically integrating the variousfunctionalities of the multiple layers into a single layer can increasethe reliability, robustness and fault tolerance functions and canimprove storage capacity, power consumption, and latency of the overallsystem. For example, solid state drive hot spots can be reduced. As themanagement system 204 can map across all of the solid state drives usingphysical addressing, usage can be spread over the devices.

Client 206 can interact with the storage system 200 using storagerequests that include fields that address layers 104, 106, 108, 110,112, 120, 122, 124, 126 and/or 128. Depending on the embodiment, client206 can issue a file system storage request, block storage request,object storage request, database storage request or structured storagerequest. These requests can be translated from a set of request fieldsto a set of return values. The request fields can address one or morelayers 104, 106, 108, 110, 112, 120, 122, 124, 126 and/or 128. In someembodiments, the fields can be addressed implicitly, such as throughprior settings. For example, a RAID stripe can be implicated because theread request occurs over several storage devices due to the RAID setup.Other layers 104, 106, 108, 110, 112, 120, 122, 124, 126 and/or 128 canbe directly addressed or managed.

For example, client 206 can create a read block storage request thatprovide fields relating to file system layer 112, snapshot/clone layer110, and mapping 126 layer. The request includes fields requesting astorage node (NODE), logical unit number (LUN), snapshot number (SNAP),clone number (CLONE) and logical block address (LBA), which can bewritten as Read(NODE, LUN, SNAP, CLONE, LBA). The client 206 can sendthe read block storage request to the management system 204. Themanagement system 204 can translate the read block storage request intoa physical address and/or attributes (e.g. ECC, etc.). The managementsystem 204 can then request the data from the storage hardware 202 usingthe physical address and/or other attributes. The storage hardware 202can address the individual solid state drive hardware 130 and return thedata to the management system 204. The management system 204 can returnthe data to the client 206. Other request types can be used such asupdate, delete, clone, snapshot, etc. For example, a file system updaterequest can include fields of NODE, storage volume (VOL), file system(FILESYS), file identifier (FILEID), stream (STREAM), SNAP, CLONE andLBA and data to store (DATA), which can be written as Update(NODE, VOL,FILESYS, FILEID, STREAM, SNAP, CLONE, LBA, DATA).

When operating as upon requests from client 206, management system 204can be viewed as a translation system between client requests andstorage hardware 202. FIG. 3 shows this abstraction of a system fortranslating storage requests into physical addresses in accordance withat least one embodiment. Client 206 can be similar or the same as client302. Management system 204 can be similar or the same as translationsystem 304. Storage hardware 306 can be similar or the same as physicalstorage 306. Solid state drive hardware 130 can be similar or the sameas storage device 308. Client 302, translation system 304 and physicalstorage can be integrated into one enclosure, be spread across multiplecomputing resources or a combination thereof.

Client 302 can make logical storage requests 310 that are translated bytranslation system 304 into physical storage requests 312 that are usedto provide return data 314 from physical storage 306. Client 302 cancreate logical storage request 310 that includes one or more fields thatidentify logical constructs supported by translation system 304.Translation system 304 can use the fields in the request to determine atranslation from logical storage request 310 to physical storage request312. The translation can result in a value and/or attributes that can beused to create physical storage request 312. Physical storage request312 can be sent to physical storage 306 to retrieve return data 314 fromone or more storage devices 308 (where the ellipsis in FIG. 3 shows thatmore than 3 storage devices may be used). The translation system 304 canthen provide the return data 314 to the client 302. In some embodiments,the return data 314 from physical storage 306 can be modified by thetranslation system 304 to meet response requirements of the client 302.In one embodiment, the translation system 304 can be compatible with anoperating system (e.g. OS Interface 114 in FIG. 1) such that it operatesas a drop-in replacement for individual storage layers (layer 104, 106,108, 110, 112, 114 and 116 in FIG. 1).

In one example, the translation system 304 causes the physical storage306 to appear as a block device. A block device request, such asRead(NODE, LUN, SNAP, CLONE, LBA) can be sent to translation system 304.The translation system 304 can use the fields of NODE, LUN, SNAP, CLONEand LBA to determine a physical storage request 312, such asRead(Physical Address, ECC, Length). In response to the physical storagerequest 312, one or more storage devices 308 can retrieve data fromstorage and provide return data 314 to the translation system 304. Thetranslation system 304 can provide return data 314 to the client 302.The translation system 304 can format and/or manipulate the data tosatisfy client 302 requirements.

The translation system 304 can include a map data structure thatdescribes the translation from logical storage request 310 into physicalstorage request 312. The map structure can be traversed from root nodeto a leaf node based on the fields provided in the request. FIG. 4 showsan example of node 402 and its structure that can be used to translatefield information into a node entry 410 or 412 that results in a nextnode pointer or a physical address.

A map structure for translating from logical storage request 310 (e.g.logical storage request 310 from FIG. 3) into physical storage request312 can include multiple nodes 402, including a root node, inner nodeand leaf node. A map traversal can start at a root node, travel throughzero or more inner nodes and stop when a leaf node is reached. At eachnode, a decision of which node is next can be made based on one or morefield values or parts thereof from logical storage request using headerinformation 404 in the node. When a leaf node is reached, a physicaladdress can be determined along with other attributes. The physicaladdress can then be used to complete the logical storage request byperforming an operation to satisfy the logical storage request 310 atthe physical address (e.g. a read operation, update operation, deleteoperation, etc.).

A node 402 can describe a step in the translation from a logical storagerequest 310 (e.g. logical storage request 310 from FIG. 3) into physicalstorage request 312. A node can comprise a header 402, a hashed storagearea 412, hashed entries 410, sorted storage area 408 and sorted entries412. A header 402 can contain metadata that describes the structure andaccess of data within the node. For example, the header 402 can containa field key identifier 414 that indicates which field is used as a keyin the node, a field mask 416 indicating which bits of the fieldidentifier are to be used in matching the key or computing an index, abitmap 418 indicating which data must be retrieved from a previous cloneand other attributes 420 as needed, such as an indication of whichvalues are stored in the hashed storage area and sorted storage area.

Entries 410 and 412 can be organized by a key defined in the header 402and store a pointer to a next node (in the case of a root or inner node)as shown or value (in the case of a leaf node) (not shown). In someembodiments, hashed entries 401 do not include a key in the entrybecause the entries are indexed by the key. The sorted entries 412 caninclude a key section to aid in comparison during search of the sortedentries 412. In one embodiment, a sorted entry 412 can include a key,flags that can provide exception information (e.g. that a prior snapshotshould be used), other attributes and a pointer to a next node (in thecase of a root or inner node) as shown or value (in the case of a leafnode) (not shown).

When used by a translation module, a node can be used to step toward atranslation of a field provided in the logical storage request to aphysical address. For example, a translation module accesses node 402during a logical storage request where LBA=62. The header 402 indicatesthat the entire LBA field should be used as the key. The header 404 alsoindicates that the range of the hashed storage area 412 begins at 0 withfour entries, which is not within the range of LBA=62. The header bitmap418 indicates that the values are within this branch of the map. Using avalue of 62 as the key, the entries 412 of the sorted storage area 408are searched for key equal to 62. The key is found and a pointer 428 isfollowed to a next node.

The root node, inner node and leaf node can include differentinformation and/or have a different structure. A root node can be astarting node for the traversal of the map. The root node can includeinformation in its header 402 to describe which field(s) or partsthereof can be used to determine a pointer to a next node. A root nodecan also include extra information not found in other nodes, such asinformation about a logical storage device, such as a basis in a priorsnapshot or clone. An inner node can be linked to from a root node orother inner node. The inner node can also include information in itsheader 402 to describe which field(s) or parts thereof can be used todetermine a pointer to a next node. A leaf node can be linked to by aroot node or other inner node.

While the node has been described as having a hybrid structure of withhashed storage area 412 and sorted storage area 408, it should berecognized that individual nodes can be composed of either or bothstructures. In some embodiments, a map uses only nodes having only onestorage type. Sizes of the hashed storage area 412 and sorted storagearea 408 can be different and vary from node to node depending onoptimizations applied, individual node settings, global node settingsand node storage needed.

With some fields, a nearest neighber can satisfy a request for aspecified field. In some embodiments, a snapshot uses nearest neighboranalysis to reduce storage costs of unchanged data over one or moresnapshots. For example, a translation module accesses node 402 during alogical storage request where SNAP=90. The header 402 indicates that theentire SNAP field should be used as the key and that nearest previousneighbor analysis should be used (in some embodiments, the nearestneighbor setting is implicit when searching snapshots). The header 404also indicates that the range of the hashed storage area 412 begins at 0with four entries, which is not within the range of SNAP=90. The headerbitmap 418 indicates that the values are within this branch of the map.Using a value of 90 as the key, the entries 412 of the sorted storagearea 408 are searched for key equal to 90. As the search ends with nomatch, but a key of 62 exists previous to the SNAP value of 90, the SNAPvalue of 62 can be used. The difference between SNAP values of 90 and 62can indicate that no change to the logical storage device has occurredsince a 62nd snapshot.

A representation of the nearest neigbhor analysis can be visualized inthree dimensional space as a relationship between unchanged informationbetween snapshots. FIG. 5 shows a diagram of translating a storagerequest into physical addresses using a closest match snapshot inaccordance with at least one embodiment. The x-axis represents a logicalunit number. The y-axis represents a logical block address. The z-axisrepresents a snapshot taken of a data plane on the x and y axis. Soliddot 506 and 508 represent data written during the selected snapshot.Hollow dot 502 and 504 represent data that was not written during theselected snapshot.

A nearest prior neighbor analysis allows a snapshot to share data withprior snapshots. For example, a client (e.g. client 302 from FIG. 3) canprovide a request of Read(LUN=2, LBA=1, SNAP=2). The translation system(e.g. translation system 304 from FIG. 3) can traverse the map throughnodes that represent LUN=2 and LBA=1. Upon arriving at a node (e.g. node402 from FIG. 4) that represents SNAP, an entry where SNAP=2 is notfound. However, a nearest neighbor entry where SNAP=0 is found. In theembodiment shown, the entry for SNAP=0 can be used because no changeshave occurred since SNAP=0 where LUN=2 and LBA=1. This lack of changecan be similar or equivalent to the path shown from hollow dot 502through hollow dot 504 to solid dot 506. As no changes have been made toLUN=2 and LBA=1 since SNAP=0, a path is traced from hollow dot 502 untilsolid dot 506 is reached indicating new data at 506.

On the other hand, as new data was writted at LUN=2, LBA=1, SNAP=2 asindicated by solid dot 508, an exact match can occur and no tracing backto previous snapshots is required.

FIGS. 6 to 9 show a diagram of an embodiment of a map at a point in timeand are used to help describe operations that can be performed on and/orwith the map. The operations have been simplified to use LUN, LBA andSNAP in FIGS. 6-8C and LUN, LBA, SNAP with CLONE awareness in FIG. 9.The operations include search, update, delete and search with CLONEawareness. The data structures and processes described can beaccomplished with the use of the systems shown in FIG. 3, includingclient 302, translation system 304 and physical storage 306. The mapdata structure can be contained in storage in translation system 304and/or physical storage 306. The operations described can be performedby the translation system 304 in response from a request provided byclient 302.

Many request types from a client (e.g. client 302 in FIG. 3) require asearch, such as a read request. FIG. 6 shows a diagram of a map fortranslating storage requests into physical addresses that can be used ina search. The map contains a root node 602 with root entries 610, 612,614 and 616. Root node entry 614 contains attributes 618 and one or moreother values 620 that include a pointer to inner node 1 604. Inner node1 includes inner node entries 622, in which one inner node entry 622points to inner node 2 606. Inner node 2 also contains inner nodeentries 622, one of which contains a pointer to leaf node 608. Leaf node608 contains a hashed storage area 626 and linear sorted storage area628. In the hashed storage area 626, two free hashed entries 630 and 634exist. Hashed entry 632 contains a key (e.g. SNAP=6, LBA=32, LEN=231)and value. In some embodiments, the LEN value can be used to determinewhich LBA addresses are covered by the entry (e.g., ADDR=32, LEN=231covers addresses from 32 to 263 which is 32+231). In the linear sortedstorage area 628, two free sorted entries 636 and 644 exist and sortedentries 638, 640 and 642 are found.

In one example, a client may request a Read(LUN=2, LBA=37, SNAP=3) andthe translation system can translate the request to a return value of(ADDR=40, LEN=240). After receipt of the client request, the translationsystem can start the search at root node 602. After reading a header ofroot node 602, the translation system can determine that the root nodeis a hashed storage area based on LUN number. Using a hash obtained fromthe root node header, where hash=LUN, the translation system can accessthe entry for LUN 2 614. Attributes about LUN 2 can be found in anattribute section of entry 614. The translation system can use a nodepointer stored in among values to find the next node, inner node 1 604.Upon arrival at inner node 1 604, the translation system can read theheader for inner node 1 604. The header can indicate that inner node 1is also hash based with the LBA field used using the bits 4-5 of the LBAfield (LBA=37 decimal=100101 binary) which are 10 binary or 2 decimal.As there are four entries 622, the value can be modulo 4, which resultsin 2. Therefore entry 2 622 is accessed in inner node 1 and thecorresponding node pointer is followed to inner node 2 606. The headernode of inner node 2 can indicate that bits 2-5 of LBA field (100101binary) are used to index into the inner node 2 606. As there are fourentries 622, the index can be modulo 4 (1001 binary is 9 decimal) theresulting entry 622 at index 1 can be accessed and followed to leaf node608.

When the translation system accesses the leaf node 608, the header canindicate a search using the LBA offset (LBA=100101 binary). As therequest indicates an LBA offset=1 and four entries, 1 modulo 4 is one.Entry 632 can be accessed to determine if it is a match. As entry 632 isnot a match, and a snapshot can be used with closest neighbor analysis(as described above, e.g. FIG. 5), the linear sorted storage area 628can be searched for a nearest neighbor which is entry 640.

Entry 640 provides a return value of (ADDR=40, LEN=240). This returnvalue can then be used to access physical storage using the address andlength. The resulting data from physical storage can be returned to theclient.

An update operation can act as a search operation to determine thecorrect node for the update and then an update when the node is reached.An update can update an entry if it exists or insert an entry if theentry does not exist. If the entry exists, the entry can be overwrittenand no further action is necessary. However, if an insert is necessary,a node can become full and the node can be separated into to two parts.The two parts can then be connected to a new parent node which takes theplace of the old prior node before splitting. FIG. 7A shows a diagram ofa map for translating storage requests into physical addresses before anupdate. FIG. 7B shows a diagram of a map for translating storagerequests into physical addresses after an update. FIG. 7C shows adiagram of a map for translating storage requests into physicaladdresses after update and full node split.

In FIG. 7A, the map contains a root node 702 with root entries 710, 712,714 and 716. Root node entry 714 contains attributes 718 and one or moreother values 720 that include a pointer to inner node 1 704. Inner node1 includes inner node entries 722, in which one inner node entry 722points to inner node 2 706. Inner node 2 also contains inner nodeentries 722, one of which contains a pointer to leaf node 708. Leaf node708 contains a hashed storage area 726 and linear sorted storage area728. In the hashed storage area 726, hashed entries 730, 732, 734 and736 exist. In the linear sorted storage area 728, hashed entries 738,740 and 742 exist, while entry 744 is empty.

The client can request an Insert(LUN=2, LBA=37, SNAP=6, Data). Afterreceipt of the client request, the translation system can start thesearch at root node 702. After reading a header of root node 702, thetranslation system can determine that the root node is a hashed storagearea based on LUN number. Using a hash obtained from the root nodeheader, where hash=LUN, the translation system can access the entry forLUN 2 714. Attributes about LUN 2 can be found in an attribute sectionof entry 714. The translation system can use a node pointer stored inamong values to find the next node, inner node 1 704. Upon arrival atinner node 1 704, the translation system can read the header for innernode 1 704. The header can indicate that inner node 1 is also hash basedwith the LBA field used using the bits 4-5 of the LBA field (LBA=37decimal=100101 binary) which are 10 binary or 2 decimal. As there arefour entries 722, the value can be modulo 4, which results in 2.Therefore entry 2 722 is accessed in inner node 1 and the correspondingnode pointer is followed to inner node 2 706. The header node of innernode 2 can indicate that bits 2-5 of LBA field (100101 binary) are usedto index into the inner node 2 706. As there are four entries 722, theindex can be modulo 4 (1001 binary is 9 decimal) the resulting entry 722at index 1 can be accessed and followed to leaf node 708.

When the translation system accesses the leaf node 708, the header canindicate a search using the LBA offset (LBA=100101 binary). As therequest indicates an LBA offset=1 and four entries, 1 modulo 4 is one.Entry 732 can be accessed to determine if it is a match. As entry 732 isnot a match and a collision, the sorted storage area 728 can besearched. As the requested insert does not have a match in the sortedstorage area 728, the translation system can perform the insert betweenentries 738 and 740 followed by an update of original entry 732 in thehashed storage area. The result of the insert can be seen in FIG. 7B.

Due to the insert performed in FIG. 7B, the leaf node 708 in is full. Asa result, the leaf node can be split into two nodes 708 and 756 as seenin FIG. 7C to allow for further future inserts.

After determining the leaf node 708 is full, the translation system cancreate two additional nodes. A new inner node 748 can receive thepointer from parent inner node 2 706. The new node can be created with ahead that identifies that LBA offesets of 0 or 1 are located in leafnode 708 and LBA offsets of 2 or 3 are located in leaf node 756. Entries730, 732, 738 and 746 can be reordered in leaf node 708. Entries 734,736, 742 and 746 can be copied to leaf node 756 and ordered.

An delete operation can act as a search operation to determine thecorrect node for the delete operation and then an delete an entry whenthe node is reached. If the entry exists and the node hosting thedeleted entry does not fall below an entry threshold as a result of thedelete, the entry can be emptied and no further action is necessary.However, if a delete is performed, a node can fall below an entrythreshold and require further action. In some cases, the node can bemerged with a neighboring node. In other cases, the map may requirerebalancing. FIG. 8A shows a diagram of a map for translating storagerequests into physical addresses in accordance with at least oneembodiment. FIG. 8B shows a diagram of a map for translating storagerequests into physical addresses after three deletions in accordancewith at least one embodiment. FIG. 8C shows a diagram of a map fortranslating storage requests into physical addresses after threedeletions and a merge in accordance with at least one embodiment.

In FIG. 8A, the map contains a root node 802 with root entries 810, 812,814 and 816. Root node entry 814 contains attributes 818 and one or moreother values 820 that include a pointer to inner node 1 804. Inner node1 includes inner node entries 822, in which one inner node entry 822points to inner node 2 806. Inner node 2 also contains inner nodeentries 822, one of which contains a pointer to inner node 3 848. Innernode 3 848 contains inner node entries 850 and 852 that point to leafnodes 808 and 842. Leaf nodes 808 and 842 contain a hashed storage area826 and linear sorted storage area 828. In the storage areas 826 of node808, entries 830, 832, 838 and 846 exist. In the storage areas 826 ofnode 856, entries 834, 836, 842 and 846 exist.

The client can request a Delete(LUN=2, LBA=36, SNAP=0). After receipt ofthe client request, the translation system can start the search at rootnode 802. After reading a header of root node 802, the translationsystem can determine that the root node is a hashed storage area basedon LUN number. Using a hash obtained from the root node header, wherehash=LUN, the translation system can access the entry for LUN 2 814.Attributes about LUN 2 can be found in an attribute section of entry814. The translation system can use a node pointer stored in amongvalues to find the next node, inner node 1 804. Upon arrival at innernode 1 804, the translation system can read the header for inner node 1804. The header can indicate that inner node 1 is also hash based withthe LBA field used using the bits 4-5 of the LBA field (LBA=37decimal=100100 binary) which are 10 binary or 2 decimal. As there arefour entries 822, the value can be modulo 4, which results in 2.Therefore entry 2 822 is accessed in inner node 1 and the correspondingnode pointer is followed to inner node 2 806. The header node of innernode 2 can indicate that bits 2-5 of LBA field (100100 binary) are usedto index into the inner node 2 806. As there are four entries 822, theindex can be modulo 4 (1001 binary is 9 decimal) the resulting entry 822at index 1 can be accessed and followed to inner node 3 848. At innernode 3 848, the header can indicate that LBA bit field 1 (100100) can beused to determine whether to go to an entry that leads to leaf node 808(for 0) or an entry that leads to leaf node 856 (for 1). As the value iszero, the pointer for leaf node 808 is followed. The leaf node canindicate that bit 0 of the LB field can be used as an index into thehashed storage area (LBA=100100). As entry 830 does not match the deleterequest, the linear sorted area can be examined for the entry which isfound at entry 838.

After being found through a search, entry 838 can be deleted from theleaf node 808. In the embodiment shown, entry 846 is copied over 838 andthe old entry 846 location is made into an empty entry 844. As a minimumthreshold of three entries exist in each leaf node, no further action isrequired.

With a few more deletions, the client can cause leaf node 808 to fallbelow a minimum threshold of occupied entries. For example, the clientcan further request Delete(LUN=2, LBA=38, SNAP=2) which is entry 846 andDelete(LUN=2, LBA=37, SNAP=4) which is entry 832. The result of thedeletions can be seen in FIG. 8B. After the deletion of entry 832, thethreshold of three entries in a leaf node has been crossed. As a result,the translation system can examine the parent node of leaf 808 (e.g.inner node 3 848) for nodes that can be merged with leaf node 808.

The translation system can determine that node 856 can be merged withnode 808 because of a shared significant bit (i.e. bit 1 of LBA) atinner node 3 848 and a combined node would not exceed a full threshold.Entries from node 856 can use insert operations into node 808 using anew bit mask of bits 0-1 of LBA that represent the merged node. Node 856can then be retired and/or deleted. Further, as inner node 3 848 now hasonly one child node 808, inner node 3 848 can be replaced by child node808. Inner node 3 848 can also be retired and/or deleted. The resultingmap after three deletions can be seen in FIG. 8C.

Cloning information can be used in conjunction with bitmap informationin a node header to indicate which data is in the current branch andwhich cloned data is from the original cloned information. A clone canbe created from a snapshot, but unlike a snapshot, the clone can befurther modified. The clone uses a different branch than the snapshot(e.g., LUN), but the clone retains links to the old information throughthe use of bitmaps in nodes that indicate which information isoverwritten and which information is original from the other branch.When traversing the map of a clone, a bitmap can indicate that anoriginal branch, from which the clone was made, should be traversedinstead to find the information. Thus, a bitmap can cause the search tobegin again from the root node using new parameters for the search thatdiffer from the request provided by the client (e.g. searching theclone's originating LUN).

FIG. 9 shows a diagram of a map for translating storage requests intophysical addresses using cloning information. For example, a branchincluding nodes 930 and 932 in LUN 3 was cloned from LUN 2. A bitmap innode 932 indicates with a value of “1001” that a first and last entry ina leaf node were overwritten, but that the second and third nodes arestill original data from the cloned information. When the bitmap isencountered by the translation system, the translation system can returnto the root node using the clone LUN and determine the parent LUN andSNAP. The translation system can then use the parent LUN and SNAP inplace of the clone information to track down the original informationthat was not overwritten by the clone.

In the example shown, a Search(LUN=3, LBA=37, SNAP=0) is requested by aclient. The translation system determines a hash function from theheader of the root node 902, which indicates that the hash function=LUN.The entry of LUN 3 916 is accessed and the pointer in value 920 isfollowed to inner node 930. Upon arrival at inner node 930, thetranslation system can read the header for inner node 930. The headercan indicate that inner node 930 is also hash based with the hash basedon bits 4-5 of the LBA field (LBA=37 decimal=100101 binary) which are 10binary or 2 decimal. As there are four entries 922, the value can bemodulo 4, which results in 2. Therefore entry 2 922 is accessed in innernode 1 and the corresponding node pointer is followed to inner node 932.The header node of inner node 2 can indicate that bits 2-5 of LBA field(100101 binary) are used to index into the inner node 2 932. As thereare four entries 922, the index can be modulo 4 (1001 binary is 9decimal) the resulting index 1 can be used to access entry 1 922, whichcontains a bitmap value. When the bitmap is compared against the remaingtwo digits of the LBA field (100101), the 0 at place 1 (1001) indicatesthat the clone data is unchanged from the original snapshot.

As the clone data is unchanged, the translation system can return to theroot node and examine the original entry 916 for a clone snapshot basis.The translation system can read the entry attributes 918 of entry 916 todetermine that the parent LUN=2 and the parent SNAP=2. The originalSearch(LUN=3, LBA=37, SNAP=0) has the same result as Search(LUN=2,LBA=37, SNAP=2). The translation system can then traverse the map usingthe new search parameters.

Using the parent parameters to form the Search(LUN=3, LBA=37, SNAP=0),the translation system can search down the parent branch of the map forthe physical address and/or attributes. The translation systemdetermines a hash function from the header of the root node 902, whichindicates that the hash function=LUN. The entry of LUN 2 914 is accessedand the pointer in value 920 is followed to inner node 904. Upon arrivalat inner node 904, the translation system can read the header for innernode 904. The header can indicate that inner node 904 is also hash basedwith the hash based on bits 4-5 of the LBA field (LBA=37 decimal=100101binary) which are 10 binary or 2 decimal. As there are four entries 922,the value can be modulo 4, which results in 2. Therefore entry 2 922 isaccessed in inner node 904 and the corresponding node pointer isfollowed to inner node 906. The header node of inner node 906 canindicate that bits 2-5 of LBA field (100101 binary) are used to indexinto the inner node 906. As there are four entries 922, the index can bemodulo 4 (1001 binary is 9 decimal) the resulting index 1 can be used toaccess entry 1 922, which points to leaf node 908.

When the translation system accesses the leaf node 908, the header canindicate a search using the LBA offset (LBA=100101 binary). As therequest indicates an LBA offset=1 and four entries, 1 modulo 4 is one.Entry 926 can be accessed to determine if it is a match, which it is.The value in entry 926 of address 32 and length 231 can be used toaccess physical memory.

Various operations described herein can be implemented on computersystems, which can be of generally conventional design. FIG. 10 is asimplified block diagram illustrating a representative computer system1000. In various embodiments, computer system 1000 or similar systemscan implement a client 206 (e.g., any of systems 114, 116) or a server(e.g., management system 204, storage hardware 202).

Computer system 1000 can include processing unit(s) 1005, storagesubsystem 1010, input devices 1020, output devices 1025, networkinterface 1035, and bus 1040.

Processing unit(s) 1005 can include a single processor, which can haveone or more cores, or multiple processors. In some embodiments,processing unit(s) 1005 can include a general-purpose primary processoras well as one or more special-purpose co-processors such as graphicsprocessors, digital signal processors, or the like. In some embodiments,some or all processing units 1005 can be implemented using customizedcircuits, such as application specific integrated circuits (ASICs) orfield programmable gate arrays (FPGAs). In some embodiments, suchintegrated circuits execute instructions that are stored on the circuititself. In other embodiments, processing unit(s) 1005 can executeinstructions stored in storage subsystem 1010.

Storage subsystem 1010 can include various memory units such as a systemmemory, a read-only memory (ROM), and a permanent storage device. TheROM can store static data and instructions that are needed by processingunit(s) 1005 and other modules of electronic device 1000. The permanentstorage device can be a read-and-write memory device. This permanentstorage device can be a non-volatile memory unit that storesinstructions and data even when computer system 1000 is powered down.Some embodiments of the invention can use a mass-storage device (such asa magnetic or optical disk or flash memory) as a permanent storagedevice. Other embodiments can use a removable storage device (e.g., afloppy disk, a flash drive) as a permanent storage device. The systemmemory can be a read-and-write memory device or a volatileread-and-write memory, such as dynamic random access memory. The systemmemory can store some or all of the instructions and data thatprocessing unit(s) 1005 need at runtime.

Storage subsystem 1010 can include any combination of computer readablestorage media including semiconductor memory chips of various types(DRAM, SRAM, SDRAM, flash memory, programmable read-only memory) and soon. Magnetic and/or optical disks can also be used. In some embodiments,storage subsystem 110 can include removable storage media that can bereadable and/or writeable; examples of such media include compact disc(CD), read-only digital versatile disc (e.g., DVD-ROM, dual-layerDVD-ROM), read-only and recordable Blue-Ray® disks, ultra densityoptical disks, flash memory cards (e.g., SD cards, mini-SD cards,micro-SD cards, etc.), magnetic “floppy” disks, and so on. The computerreadable storage media do not include carrier waves and transitoryelectronic signals passing wirelessly or over wired connections.

In some embodiments, storage subsystem 1010 can store one or moresoftware programs to be executed by processing unit(s) 1005, such as anoperating system, a browser application, a mobile app for accessing anonline content management service, a desktop application for accessingthe online content management service, and so on. “Software” refersgenerally to sequences of instructions that, when executed by processingunit(s) 1005 cause computer system 1000 to perform various operations,thus defining one or more specific machine implementations that executeand perform the operations of the software programs. The instructionscan be stored as firmware residing in read-only memory and/orapplications stored in non-volatile storage media that can be read intovolatile working memory for execution by processing unit(s) 1005.Software can be implemented as a single program or a collection ofseparate programs or program modules that interact as desired. Fromstorage subsystem 1010, processing unit(s) 1005 can retrieve programinstructions to execute and data to process in order to execute variousoperations described herein.

A user interface can be provided by one or more user input devices 1020and one or more user output devices 1025. Input devices 1020 can includeany device via which a user can provide signals to computing system1000; computing system 1000 can interpret the signals as indicative ofparticular user requests or information. In various embodiments, inputdevices 1020 can include any or all of a keyboard, touch pad, touchscreen, mouse or other pointing device, scroll wheel, click wheel, dial,button, switch, keypad, microphone, and so on.

User output devices 1025 can include any device via which computersystem 1000 can provide information to a user. For example, user outputdevices 1025 can include a display to display images generated bycomputing system 1000. The display can incorporate various imagegeneration technologies, e.g., a liquid crystal display (LCD),light-emitting diode (LED) including organic light-emitting diodes(OLED), projection system, cathode ray tube (CRT), or the like, togetherwith supporting electronics (e.g., digital-to-analog oranalog-to-digital converters, signal processors, or the like). Someembodiments can include a device such as a touchscreen that function asboth input and output device. In some embodiments, other user outputdevices 1025 can be provided in addition to or instead of a display.Examples include indicator lights, speakers, tactile “display” devices,printers, and so on.

Network interface 1035 can provide voice and/or data communicationcapability for computer system 1000. In some embodiments, networkinterface 1035 can include radio frequency (RF) transceiver componentsfor accessing wireless voice and/or data networks (e.g., using cellulartelephone technology, advanced data network technology such as 3G, 4G orEDGE, WiFi (IEEE 802.11 family standards), or other mobile communicationtechnologies, or any combination thereof), GPS receiver components,and/or other components. In some embodiments, network interface 1035 canprovide wired network connectivity (e.g., Ethernet) in addition to orinstead of a wireless interface. Network interface 1035 can beimplemented using a combination of hardware (e.g., antennas,modulators/demodulators, encoders/decoders, and other analog and/ordigital signal processing circuits) and software components.

Bus 1040 can include various system, peripheral, and chipset buses thatcommunicatively connect the numerous components of computing system1000. For example, bus 1040 can communicatively couple processingunit(s) 1005 with storage subsystem 1010.

Bus 1040 can also connect to input devices 1020 and output devices 1025.Bus 1040 can also couple computing system 1000 to a network throughnetwork interface 1035. In this manner, computing system 1000 can be apart of a network of multiple computer systems (e.g., a local areanetwork (LAN), a wide area network (WAN), an intranet, or a network ofnetworks, such as the Internet.

Some embodiments include electronic components, such as microprocessors,storage and memory that store computer program instructions in acomputer readable storage medium. Many of the features described in thisspecification can be implemented as processes that are specified as aset of program instructions encoded on a computer readable storagemedium. When these program instructions are executed by one or moreprocessing units, they cause the processing unit(s) to perform variousoperation indicated in the program instructions. Examples of programinstructions or computer code include machine code, such as is producedby a compiler, and files including higher-level code that are executedby a computer, an electronic component, or a microprocessor using aninterpreter.

Through suitable programming, processing unit(s) 1005 can providevarious functionality for computing device 1000. For example, in amobile computing device, processing unit(s) 1005 can execute anoperating system capable of communicating with storage system 100. In adesktop computing device, processing unit(s) 1005 can execute anoperating system and a desktop application program that presents aninterface to storage system 100; in some embodiments, this interface maybe integrated with an interface to a file system maintained by theoperating system. In some embodiments, processing unit(s) 1005 canexecute an application that provides the ability to retrieve and displaydata from sources such as storage system 100.

In some embodiments, computer system 1000 or a similar system can alsoimplement operating system 302, translation system 304 or physicalstorage 306. In such instances, a user interface may be located remotelyfrom processing unit(s) 1005 and/or storage subsystem 1010; similarly,storage subsystem 1010 or portions thereof may be located remotely fromprocessing unit(s) 1005. Accordingly, in some instances, variouscomponents of computer system 1000 need not be physically located in anyparticular proximity to each other.

It will be appreciated that computer system 1000 is illustrative andthat variations and modifications are possible. Computer system 1000 canhave other capabilities not specifically described here (e.g., router,email reporting, mobile phone, global positioning system (GPS), powermanagement, one or more cameras, various connection ports for connectingexternal devices or accessories, etc.). Further, while computer system1000 is described with reference to particular blocks, it is to beunderstood that these blocks are defined for convenience of descriptionand are not intended to imply a particular physical arrangement ofcomponent parts. Further, the blocks need not correspond to physicallydistinct components. Blocks can be configured to perform variousoperations, e.g., by programming a processor or providing appropriatecontrol circuitry, and various blocks might or might not bereconfigurable depending on how the initial configuration is obtained.Embodiments of the present invention can be realized in a variety ofapparatus including electronic devices implemented using any combinationof circuitry and software.

FIGS. 11 to 15 show processes that can be accomplished through the useof system recited herein (e.g. client 302, translation system 304 andphysical storage 306 from FIG. 3). The processes include search, update,delete, clone and snapshot. The data structures and processes describedcan be accomplished with the use of the systems shown in FIG. 3,including client 302, translation system 304 and physical storage 306.The map data structure can be contained in storage located intranslation system 304 and/or physical storage 306. The operationsdescribed can be performed by the translation system 304 in response toa request provided by client 302.

FIG. 11 shows an illustrative example of a process 1100 that may be usedto search a map for translating a storage request into physicaladdresses in accordance with at least one embodiment. A translationsystem can receive a request to retrieve data from a client. The requestcan include fields that describe a logical placement of the data. Inblock 1102, the translation system can start a root node. In block 1104,the translation system reads header information from the root node todetermine a key (e.g. from one or more of the fields) for searching thenode and whether a hashed search or sorted search should be performed.In block 1106, a search type is selected based on the headerinformation. If a sorted search is selected, in block 1114 a searchmethod can be used (e.g., binary search) to find a matching entry withthe key in a sorted storage area. In block 1108, if the search selectedis a hashed-based search, an index into the hash storage area can bedetermined and/or computed based on the identified key. An entry can beaccessed based on the computed hash. In block 1110, any flags present inthe entry can be read (e.g. a bitmap as described above). If the entryis not found in block 1112, a fallback binary search can be performed inblock 1114 if needed.

The binary search from 1114 or hash-based search from block 1112 canexamine a selected entry to determine if there is a matching entry inblock 1116. In block 1118, if no match, a closest neighbor may be usedif appropriate (e.g. snapshot). If, in block 1118, the closest neighboris not appropriate, then in block 1120, the entry does not exist. Insome embodiments, if the entry does not exist, the translation systemcan decide whether to search the parent in block 1122. However, if anentry does match from block 1116 or a closest neighbor is appropriatefrom block 1118, then the entry can be reviewed in block 1122 todetermine if a parent entry of clone should be used (e.g. a bitmapindicated a parent snapshot should be used). In block 1128, if theparent entry should be used then the translation system uses the parententry and returns to the root node in block 1002. If not, in block 1124,the matched entry does not reside in a leaf node, the pointer in theentry can be followed to the next node where the header is read in block1104. If in block 1124, the matched entry is a leaf node, then thetranslation system can obtain physical addresses and attributes from thematched entry in the leaf node. The physical address and attributes canthen be used to access a physical storage system to operate on data(e.g. retrieve data).

An update process can begin with a search for a matching entry, whichcan be followed by an overwrite or an insert process. If a leaf nodebecomes full, the leaf node can be split into two nodes. FIG. 12 showsan illustrative example of a process 1200 that may be used to update amap for translating a storage request into physical addresses inaccordance with at least one embodiment. In block 1202, a search for thelocation of an entry can be performed (e.g. search process 1100 fromFIG. 11). In block 1204, if an exact match is found, a node entry valuecan be updated in block 1206 which completes the process in block 1208.However, if in block 1204 a leaf node is reached and there is not anexact match, a determination is made in block 1210 whether there is roomto insert the new entry into a hashed storage area of the leaf node. Ifso, in block 1212, a new entry can be inserted into the hashed storagearea of the leaf node which completes the process in block 1208. If theentry cannot be inserted into the hashed storage area, in block 1214,the node can be examined to see if it is full. If the node is not full,in block 1216, the entry can be inserted into the sorted storage areawhich completes the process in block 1208. If the node is full, in block1218, the node can be split into two nodes in which the node entries aredivided and the header of each node can be updated to accommodate thesplit (e.g. remove a significant digit of a field or modify the modulusvalue because of the division of the entries). The old node can receivehalf of the entries and a newly created node can receive half of theentries. In block 1220, the new node can be attempted to be insertedinto the parent node, which moves to block 1210 to attempt the insert.

A delete process can begin with a search for the node to delete. If thenode can be deleted without triggering an minimum threshold of entriesin a log, the process can stop. However, if the threshold is triggered,a merging of the triggering node and a neighbor can be attempted. Ifthat is not able to work, the map can be rebalanced.

FIG. 13 shows an illustrative example of a process 1300 that may be usedto delete an entry in a map for translating a storage request intophysical addresses in accordance with at least one embodiment. In block1302, a search for the location of an entry can be performed (e.g.search process 1100 from FIG. 11). In block 1304, if an entry is notfound, the delete process cannot move forward and is complete in block1314. In block 1306, if the node entry is found, the node type can bedetermined. In block 1308 node entries in hashed storage areas can bemade empty. In block 1312, node entries in sorted storage areas can bedeleted from the sorted list. After the deletion in block 1312 or madeempty in block 1308, the node can be examined to see if the node has aninsufficient number of node entries such that an empty threshold isreached. If the threshold is not reached, the process can be completedin node 1314. However, if the node has reached an empty threshold, inblock 1316 neighboring nodes can be examined to see if a merging ofnodes is possible. If merging is not possible, the translation systemcan cause the node entries to be rebalanced in the map (e.g. using aB-Tree process, etc.) and the deletion can be completed in block 1314.However, if the node can merge with a neighbor node, in block 1318, thenode merges entries with the neighbor node to form a node with entriesand an empty node. In block 1320, the empty node is deleted from aparent node which can cause a test for an empty threshold of in block1310.

Creation of a clone and/or snapshot can be a simple process. In someembodiments, copying does not occur, but a number is set in a nodeattribute. In FIGS. 14 and 15 processes are shown that can createsnapshots and clones by changing values rather than copying data. In theembodiments shown, the snapshot and clone can be quickly preparedwithout a time cost related to size. As the data can remain stationary,a snapshot or clone can be created without data copy overhead.

When a clone is created, a new LUN is created to hold new data for thesnapshot. However, the cloned data remains with the old LUN as part of asnapshot. As the old data remains in its current position, retrievingthe original clone data can be accomplished by processes and systemsdescribed in connection with FIG. 9. FIG. 14 shows an illustrativeexample of a 1400 process that may be used to clone a Logical Unit (LUN)in a map for translating a storage request into physical addresses inaccordance with at least one embodiment. In block 1402, the translationsystem can create a new LUN for the clone. In block 1404, the new LUNnode can be updated with parent information of the clone (e.g. parentLUN and parent SNAP can be stored as attributes in a LUN entry), afterwhich the process can be completed in block 1406.

When a snapshot is created, a current snapshot number is incremented. Insome embodiments, this number is located in an attribute entry of a noderepresenting a LUN. FIG. 15 shows an illustrative example of a process1500 that may be used to snapshot a Logical Unit (LUN) in a map fortranslating a storage request into physical addresses in accordance withat least one embodiment. In block 1502, a target LUN node can be found(e.g. through search) and selected. In some embodiments, the LUN entryis part of a root node and is configured to store these extraattributes. In block 1504, a snapshot number attribute is increased inthe selected LUN node entry which completes the process at block 1506.

It should be recognized that in some figures only necessary portions ofa diagram, such as a map, may be shown. It should also be recognizedthat example fields and orders of fields have been given, but that otherorders and fields may be used. Processes can also be accomplished inother orderings and/or operations of processes can be performed inparallel.

While examples have been simplified and examples have been given interms of a block device, it should be recognized that the processes canbe used with other storage types (e.g. file system storage, objectstorage, database storage, structured storage, etc.).

What is claimed is:
 1. A computer implemented method, comprising: receiving a request to add data at a logical location in a storage system, the request including a set of fields describing the logical location in the storage system; traversing a map of nodes from a root node to a parent node of a first node identified as containing the logical location, wherein at least a subset of fields from the set of fields represent a key for navigating the map of nodes, each node identifying a portion of the key used to select an entry in the node, the entry identifying a next node or a physical location in the storage system; determining that adding a node entry to the first node exceeds a threshold utilization of entries; determining a revised parent portion of the key used to select parent node entries in the parent node to enable redistribution of first node entries from the first node; creating a second node; determining a revised first portion of the key to use with the first node; determining a second portion of the key to use with the second node; redistributing first node entries from the first node to the first node and the second node based at least in part on the revised parent portion of the key in the parent node; and providing entries for the first node and the second node in the parent node based at least in part on the revised parent portion of the key.
 2. The computer implemented method of claim 1, wherein traversing the map of nodes from the root node to the parent node further comprises: starting at the root node of the map of nodes describing the storage system, the map of nodes comprising a plurality of nodes, at least some of the plurality of nodes comprising a hashed portion and a sorted portion; and determining an end node of the map of nodes by traversing nodes in the map of nodes through: determining a first portion of the set of fields to use as the key to a node entry in the node; locating the node entry in the hashed portion or the sorted portion of the node based on the key; when the node entry links to a next node, following a link to the next node; and when the node entry includes a physical address, retrieving data from a storage device using the physical address.
 3. The computer implemented method of claim 1, wherein determining that the first node exceeds the threshold utilization of entries further comprises determining that a hashed portion of node storage exceeds the threshold utilization of entries.
 4. The computer implemented method of claim 1, wherein determining that the first node exceeds the threshold utilization of entries further comprises determining that a sorted portion of node storage exceeds the threshold utilization of entries.
 5. The computer implemented method of claim 1, wherein determining the revised parent portion of the key used to select the entries in the parent node further comprises dividing a parent portion of the key into a set including a more significant portion and a less significant portion.
 6. The computer implemented method of claim 5, further comprising selecting a more significant portion to use as the revised parent portion of the key.
 7. The computer implemented method of claim 5, wherein the key is represented as a set of bits.
 8. A system comprising: a storage interface configured to receive requests for data at a logical storage location; a storage system comprising a set of physical locations; and a translation system configured to form a translation of the logical storage location to a physical storage location in the set of physical locations by: receiving a request to add data at a logical location in the storage system, the request including a set of fields describing the logical location in the storage system; traversing a map of nodes to locate a first node to receive a reference to the data based at least in part on a subset of fields from the set of fields that represent a key for navigating the map of nodes; determining that the first node exceeds a threshold utilization of entries; creating a second node; rebalancing node entries from the first node to at least the first node and the second node; revising key portions used to select node entries for the first node, the second node and a parent node of the first node and the second node; providing entries for the first node and the second node in the parent node based at least in part on the revised key portions; and adding a node entry to the rebalanced node entries based at least in part on the revised key portions.
 9. The system of claim 8, wherein the storage system comprises solid state drives.
 10. The system of claim 8, wherein the set of fields represents a logical block location.
 11. The system of claim 10, wherein the logical block location further comprises an identifier of a storage node, logical unit number, snapshot number, clone number or logical block address.
 12. The system of claim 8, wherein the set of fields represents a file system location.
 13. The system of claim 12, wherein the file system location further comprises an identifier of a storage volume, file system, file, stream, snapshot number, clone number or logical block address.
 14. The system of claim 8, further comprising an operating system in communication with the storage interface.
 15. One or more non-transitory computer-readable storage media having collectively stored thereon executable instructions that, when executed by one or more processors of a computer system, cause the computer system to at least: receive a request to remove data at a logical location in a storage system, the request including a set of fields describing the logical location in the storage system; traverse a map of nodes to locate a first node including a node entry that references the data based at least in part on a subset of fields from the set of fields that represent a key for navigating the map of nodes; invalidate the node entry from the first node; determine that the first node is below a threshold utilization of entries based at least in part on invalidating the node entry; determine that a second node is a sibling of the first node and has sufficient space to merge node entries from the first node; merge node entries from the first node to the second node; remove the first node from the map of nodes; revise a parent node entry in a parent node of the first node and the second node to reflect the merged node entries; and revise key information for the second node based at least in part on the merged node entries.
 16. The non-transitory computer-readable storage media of claim 15, wherein the instructions further comprise instructions that, when executed, cause the computer system to at least: receive a second request to remove second data at a second logical location in the storage system, the second request including a second set of fields describing the second logical location in the storage system; traverse the map of nodes to locate a third node including a second node entry that references the second data based at least in part on a second subset of fields from the second set of fields that represent a second key for navigating the map of nodes; invalidate the second node entry from the third node; determine that the third node is below the threshold utilization of entries based at least in part on invalidating the second node entry; determine that a fourth node has a sibling relationship to the third node and has insufficient space to merge node entries from the first node; rebalance a node entry distribution between the third node and the fourth node in the map of nodes; and revise key information in use for the third node, the fourth node and the parent node of the third node and the fourth node based at least in part on the rebalanced node entry distribution.
 17. The non-transitory computer-readable storage media of claim 15, wherein the instructions further comprise instructions that, when executed, cause the computer system to at least: receive a second request to remove second data at a second logical location in the storage system, the second request including a second set of fields describing the second logical location in the storage system; traverse the map of nodes to locate a third node including a second node entry that references the second data based at least in part on a second subset of fields from the second set of fields that represent a second key for navigating the map of nodes; invalidate the second node entry from the third node; determine that the third node is below the threshold utilization of entries based at least in part on invalidating the second node entry; determine that the third node has no sibling relationship to a next node in the map of nodes; rebalance a node entry distribution between the third node and the parent node of the third node in the map of nodes; and revise key information in use for the third node and the parent node of the third node based at least in part on the rebalanced node entry distribution.
 18. The non-transitory computer-readable storage media of claim 15, wherein the instructions further comprise instructions that, when executed, cause the computer system to at least mark a physical location of the data as free space.
 19. The non-transitory computer-readable storage media of claim 15, wherein revising key portions in use for the second node further comprises remapping, in the second node, a hashed portion of node entries and an indexed portion of node entries.
 20. The non-transitory computer-readable storage media of claim 15, wherein the instructions further comprise instructions that, when executed, cause the computer system to at least respond to the request confirming removal of the data. 